<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head><title></title>
<link href="../style/ebook.css" type="text/css" rel="stylesheet">
</head>
<body>
<h1>Scaling Out with Dask</h1>
<p><code class="docutils literal notranslate"><span class="pre">DaskExecutor</span></code> allows you to run Airflow tasks in a Dask Distributed cluster.</p>
<p>Dask clusters can be run on a single machine or on remote networks. For complete
details, consult the <a class="reference external" href="https://distributed.readthedocs.io/">Distributed documentation</a>.</p>
<p>To create a cluster, first start a Scheduler:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># default settings for a local cluster</span>
<span class="nv">DASK_HOST</span><span class="o">=</span><span class="m">127</span>.0.0.1
<span class="nv">DASK_PORT</span><span class="o">=</span><span class="m">8786</span>

dask-scheduler --host <span class="nv">$DASK_HOST</span> --port <span class="nv">$DASK_PORT</span>
</pre>
</div>
</div>
<p>Next start at least one Worker on any machine that can connect to the host:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>dask-worker <span class="nv">$DASK_HOST</span>:<span class="nv">$DASK_PORT</span>
</pre>
</div>
</div>
<p>Edit your <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> to set your executor to <code class="docutils literal notranslate"><span class="pre">DaskExecutor</span></code> and provide
the Dask Scheduler address in the <code class="docutils literal notranslate"><span class="pre">[dask]</span></code> section.</p>
<p>Please note:</p>
<ul class="simple">
<li>Each Dask worker must be able to import Airflow and any dependencies you
require.</li>
<li>Dask does not support queues. If an Airflow task was created with a queue, a
warning will be raised but the task will be submitted to the cluster.</li>
</ul>
</body>
</html>